Semantic MapNet: Building Allocentric Semantic Maps and Representations from Egocentric Views

We study the task of semantic mapping – specifically, an embodied agent (a robot or egocentric AI assistant) is given a tour new environment and asked to build allocentric top-down map (‘what where?’) from observations RGB-D camera with known pose (via localization sensors). Importantly, our goal neural episodic memories spatio-semantic representations 3D spaces that enable easily learn subsequent tasks in same space navigating objects seen during (‘Find chair’) answering questions about (‘How many chairs did you see house?’). Towards this goal, we present Semantic MapNet (SMNet), which consists of: (1) Egocentric Visual Encoder encodes each frame, (2) Feature Projector projects features appropriate locations on floor-plan, (3) Spatial Memory Tensor size floor-plan length×width×feature-dims learns accumulate projected features, (4) Map Decoder uses memory tensor produce maps. SMNet combines strengths (known) projective geometry representation learning. On Matterport3D dataset, significantly outperforms competitive baselines by 4.01?16.81% (absolute) mean-IoU 3.81?19.69% Boundary-F1 metrics. Moreover, show how use for ObjectNav Embodied Question Answering. Project page: https://vincentcartillier.github.io/smnet.html.